{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using NER to Map MARC Geographic Subject Headings\n",
    "*Now with a strategy for disambiguating placenames!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of my main research projects at the [ISAW Library](http://isaw.nyu.edu/library) has been geolocating subject headings of books in our collection. That is, not determining \"What a book is about?\" but \"Where is it about?\" The notebook below goes through an example workflow that takes us from a book's unique identifier in the library's catalog to a map showing geolocated subject headings. The workflow is as follows:\n",
    "\n",
    "- Retrieve a MARC record from the [NYU Library LMS](https://aleph.library.nyu.edu/)\n",
    "- Run the text of the MARC record through a named entity recognition tagger ([Stanford NER Tagger](https://nlp.stanford.edu/software/CRF-NER.html))\n",
    "- Separate out 'location' NER data\n",
    "- Query the [Geonames](http://www.geonames.org/) API for geographic coordinates for all locations\n",
    "- Map the coordinates using [Folium](http://python-visualization.github.io/folium/)\n",
    "\n",
    "For this notebook, I use the American School of Classical Studies at Athens's 1947 [\"Ancient Corinth: A guide to the excavations\"](http://www.worldcat.org/title/ancient-corinth-a-guide-to-the-excavations/oclc/10220993). (This is the fourth edition, revised by O. Broneer.) The choice of the book itself it not particularly important——but I wanted something that had a clear and obvious answer to \"Where is this book about?\" (Unsurprisingly, [Corinth](http://www.ascsa.edu.gr/index.php/excavationcorinth/).) But one that, as we will see, presents questions about ambiguity in place names as well as which place names in fact answer the research question.\n",
    "\n",
    "Going through this workflow, it becomes clear that the step \"Query the Geonames API...\" presents difficulties. The most relevant place name for the query \"corinth\", for example, is \"Corinth, TX\". When queried by relevance, \"Corinth, Greece\" only appears after 46 other Corinths. Luckily, we have other data that we can use to help determine which 'corinth' is the best choice. Here is one way of approaching the problem. In addition to coordinate information, we can also retrieve 'country' information. If we restrict our coordinate search to places that appear in the moist frequently occurring 'country' in all of the searches—here, Greece— we can get (at least for this query) more accurate results. This obviously will not work for all \"Where is this book about?\" searches, but the general principle of using additional data from Geonames json to cluster multiple terms should in some manner prove useful. The results here of Corinth, Athens, and Greece, are good enough for this simple experiment. In the next notebook, we will look at strategies for weighting—or in the case of Athens, perhaps excluding—returned place names to improve the usefulness of our geographic subject-headings map. [PJB 3.15.18]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "\n",
    "import os\n",
    "\n",
    "import xml.etree.ElementTree as ET\n",
    "import json\n",
    "import requests\n",
    "\n",
    "from collections import Counter, defaultdict\n",
    "import random\n",
    "\n",
    "from nltk.tag import StanfordNERTagger\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk import pos_tag\n",
    "from nltk.chunk import conlltags2tree\n",
    "from nltk.tree import Tree\n",
    "\n",
    "import folium\n",
    "\n",
    "from pprint import pprint\n",
    "from tqdm import tqdm #for progress bar\n",
    "\n",
    "# Constants\n",
    "isbn_valid = '0123456789xX'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set environment variable\n",
    "# Geonames requires a username to access the API but we do not want to expose personal info in code\n",
    "# Run this locally by adding USERNAME to environment variables, e.g. to .env, as follows:\n",
    "# > export USERNAME=<insert username here>\n",
    "USERNAME = os.getenv('USERNAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get MARC Records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample BSN; BSN is a code that NYU Libraries use to identify books in the collection\n",
    "# This is the BSN for... \n",
    "# \"Ancient Corinth: A guide to the excavations,\" O. Broneer, R. Carpenter, and C. H. Morgan\n",
    "\n",
    "bsns = ['003442638']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Request MARC records for each BSN\n",
    "# NB: Only works on the NYU network; I provide sample output as a string in the example\n",
    "# NB: This notebook takes the MARC XML and reduces it to a single string; it would probably be better\n",
    "# to isolate only the subject headings (and perhaps the title). Without doing this, NER is going to pick\n",
    "# up the imprint information as well.\n",
    "\n",
    "# marcs = []\n",
    "\n",
    "# for bsn in tqdm(bsns):\n",
    "#     urlstring = 'http://aleph.library.nyu.edu/X?op=publish_avail&library=nyu01&doc_num=%s' % bsn\n",
    "#     aleph_request = requests.get(urlstring)\n",
    "#     aleph_string = aleph_request.text\n",
    "#     tree = ET.fromstring(aleph_string)\n",
    "#     marc = \" \".join(ET.tostring(tree, encoding='utf8', method='text').decode('utf-8').split('\\n')).strip()\n",
    "#     marcs.append(marc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['aleph-publish:003442638    00000cam a2200421 a 4500 BK 00000cam a2200421 a 4500 003442638 OCoLC 20110421131315.0 110421s1947    gr aef        000 0 eng d  (OCoLC)10220993   CIN eng CIN OCL OCLCQ OCLCG OCLCA OCLCQ NNU   eng   e-gr---   DF261.C65 A57 1947   (OCoLC)10220993   Ancient Corinth : a guide to the excavations.   Guide to the excavations of ancient Corinth   4th ed., rev. and enl.   [Athens? : Printing Office \"Hestia\"], 1947.   127 p., [2] leaves of plates : ill., plans (some fold.) ; 21 cm.   At head of title: American School of Classical Studies at Athens.   Preface to the 4th ed. signed by Oscar Broneer.   \"The Corinth Guide, originally written by Rhys Carpenter in 1928 and revised by him in 1933, was again revised by Charles H. Morgan in 1936 ... \"--Preface to the 4th ed.   ISAW copy is from the library of Paul Åström. NyNyUAW   Corinth (Greece) Antiquities.   Excavations (Archaeology) Greece Corinth.   Greece Antiquities.   Broneer, Oscar, 1894-1992.   Carpenter, Rhys, 1889-1980. Ancient Corinth, a guide to the excavations and museums.   Morgan, Charles H. (Charles Hill), 1902-1984.   American School of Classical Studies at Athens.   Åström, Paul, former owner. NyNyUAW   Åström Collection. NyNyUAW   Online version: American School of Classical Studies at Athens. Ancient Corinth. 4th ed., rev. and enl. [Athens, \"Hestia\"] 1947 (OCoLC)609019079   Z0 ZYU   GENERAL   NYU50 NISAW Small Collection DF261.C65 A57 1947 Non-circulating available Available 1 0 N 0 SMALL 0       N7KK31FMHB56YMNSV5J7KHNK3A9VV7BNF11BQA5H6LD126BJLF']\n"
     ]
    }
   ],
   "source": [
    "marcs = ['aleph-publish:003442638    00000cam a2200421 a 4500 BK 00000cam a2200421 a 4500 003442638 OCoLC 20110421131315.0 110421s1947    gr aef        000 0 eng d  (OCoLC)10220993   CIN eng CIN OCL OCLCQ OCLCG OCLCA OCLCQ NNU   eng   e-gr---   DF261.C65 A57 1947   (OCoLC)10220993   Ancient Corinth : a guide to the excavations.   Guide to the excavations of ancient Corinth   4th ed., rev. and enl.   [Athens? : Printing Office \"Hestia\"], 1947.   127 p., [2] leaves of plates : ill., plans (some fold.) ; 21 cm.   At head of title: American School of Classical Studies at Athens.   Preface to the 4th ed. signed by Oscar Broneer.   \"The Corinth Guide, originally written by Rhys Carpenter in 1928 and revised by him in 1933, was again revised by Charles H. Morgan in 1936 ... \"--Preface to the 4th ed.   ISAW copy is from the library of Paul Åström. NyNyUAW   Corinth (Greece) Antiquities.   Excavations (Archaeology) Greece Corinth.   Greece Antiquities.   Broneer, Oscar, 1894-1992.   Carpenter, Rhys, 1889-1980. Ancient Corinth, a guide to the excavations and museums.   Morgan, Charles H. (Charles Hill), 1902-1984.   American School of Classical Studies at Athens.   Åström, Paul, former owner. NyNyUAW   Åström Collection. NyNyUAW   Online version: American School of Classical Studies at Athens. Ancient Corinth. 4th ed., rev. and enl. [Athens, \"Hestia\"] 1947 (OCoLC)609019079   Z0 ZYU   GENERAL   NYU50 NISAW Small Collection DF261.C65 A57 1947 Non-circulating available Available 1 0 N 0 SMALL 0       N7KK31FMHB56YMNSV5J7KHNK3A9VV7BNF11BQA5H6LD126BJLF']\n",
    "print(marcs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Extraction on MARC Record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/patrick/Envs/mapping-experiments-zz141tLE/lib/python3.6/site-packages/nltk/tag/stanford.py:183: DeprecationWarning: \n",
      "The StanfordTokenizer will be deprecated in version 3.2.5.\n",
      "Please use \u001b[91mnltk.tag.corenlp.CoreNLPPOSTagger\u001b[0m or \u001b[91mnltk.tag.corenlp.CoreNLPNERTagger\u001b[0m instead.\n",
      "  super(StanfordNERTagger, self).__init__(*args, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "# Setup Stanford NER Tagger\n",
    "# Ignore deprecation warning for now; we'll deal with it when the time comes!\n",
    "\n",
    "st = StanfordNERTagger('/usr/local/share/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz', \n",
    "                       '/usr/local/share/stanford-ner/stanford-ner.jar',\n",
    "                       encoding='utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Functions for putting together with inside-outside-beginning (IOB) logic\n",
    "# Cf. https://stackoverflow.com/a/30666949\n",
    "#\n",
    "# For more information on IOB tagging, see https://en.wikipedia.org/wiki/Inside–outside–beginning_(tagging)\n",
    "\n",
    "def stanfordNE2BIO(tagged_sent):\n",
    "    bio_tagged_sent = []\n",
    "    prev_tag = \"O\"\n",
    "    for token, tag in tagged_sent:\n",
    "        if tag == \"O\": #O\n",
    "            bio_tagged_sent.append((token, tag))\n",
    "            prev_tag = tag\n",
    "            continue\n",
    "        if tag != \"O\" and prev_tag == \"O\": # Begin NE\n",
    "            bio_tagged_sent.append((token, \"B-\"+tag))\n",
    "            prev_tag = tag\n",
    "        elif prev_tag != \"O\" and prev_tag == tag: # Inside NE\n",
    "            bio_tagged_sent.append((token, \"I-\"+tag))\n",
    "            prev_tag = tag\n",
    "        elif prev_tag != \"O\" and prev_tag != tag: # Adjacent NE\n",
    "            bio_tagged_sent.append((token, \"B-\"+tag))\n",
    "            prev_tag = tag\n",
    "\n",
    "    return bio_tagged_sent\n",
    "\n",
    "\n",
    "def stanfordNE2tree(ne_tagged_sent):\n",
    "    bio_tagged_sent = stanfordNE2BIO(ne_tagged_sent)\n",
    "    sent_tokens, sent_ne_tags = zip(*bio_tagged_sent)\n",
    "    sent_pos_tags = [pos for token, pos in pos_tag(sent_tokens)]\n",
    "\n",
    "    sent_conlltags = [(token, pos, ne) for token, pos, ne in zip(sent_tokens, sent_pos_tags, sent_ne_tags)]\n",
    "    ne_tree = conlltags2tree(sent_conlltags)\n",
    "    return ne_tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:03<00:00,  3.19s/it]\n"
     ]
    }
   ],
   "source": [
    "# Apply NER to each MARC record\n",
    "\n",
    "places_set = []\n",
    "\n",
    "for marc in tqdm(marcs):\n",
    "    marc_coordinates = []\n",
    "    tokenized_marc = word_tokenize(marc)\n",
    "    classified_marc = st.tag(tokenized_marc)\n",
    "    classified_marc = [item for item in classified_marc if item[0] != ''] # clean up parsing\n",
    "    ne_tree = stanfordNE2tree(classified_marc)\n",
    "\n",
    "    ne_in_sent = []\n",
    "    for subtree in ne_tree:\n",
    "        if type(subtree) == Tree: # If subtree is a noun chunk, i.e. NE != \"O\"\n",
    "            ne_label = subtree.label()\n",
    "            ne_string = \" \".join([token for token, pos in subtree.leaves()])\n",
    "            ne_in_sent.append((ne_string, ne_label))\n",
    "    \n",
    "    locations = set([tag[0] for tag in ne_in_sent if tag[1] == 'LOCATION']) # If we don't make this a set, we can use frequency info for map weight\n",
    "    \n",
    "    places_set.append(locations)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Athens', 'Corinth', 'Greece', 'Greece Antiquities', 'Greece Corinth']\n"
     ]
    }
   ],
   "source": [
    "places_set = [sorted(item) for item in places_set]\n",
    "print(places_set[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Geolocate place names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for querying geonames\n",
    "\n",
    "def geonames_query(location):\n",
    "    '''\n",
    "    queries geonames for given location name;\n",
    "    bounding box variables contain default values\n",
    "    based on: https://prpole.github.io/location-extraction-georeferencing/    \n",
    "    '''\n",
    "    # Todo\n",
    "    # - replace error handling\n",
    "\n",
    "    baseurl = 'http://api.geonames.org/searchJSON' #baseurl for geonames\n",
    "    username = USERNAME # Keep USERNAME in .env\n",
    "    json_decode = json.JSONDecoder() #used to parse json response\n",
    "\n",
    "    params = {\n",
    "        'username': username, \n",
    "        'name_equals': location,\n",
    "        'orderby': 'relevance',\n",
    "    }\n",
    "    \n",
    "    response = requests.get(baseurl, params=params)\n",
    "    response_string = response.text\n",
    "    parsed_response = json_decode.decode(response_string)\n",
    "    \n",
    "    if 'geonames' in parsed_response.keys():\n",
    "        if len(parsed_response['geonames']) > 0:\n",
    "            first_response = parsed_response['geonames'][0]\n",
    "            coordinates = (first_response['lat'],first_response['lng'])\n",
    "        else: \n",
    "            coordinates = ('','')\n",
    "    else:\n",
    "        coordinates = ('','')\n",
    "    return coordinates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('37.97945', '23.71622'), ('33.15401', '-97.06473'), ('43.20978', '-77.69306')]\n"
     ]
    }
   ],
   "source": [
    "# Build list of likely coordinates for places\n",
    "\n",
    "places_list = []\n",
    "coordinates_list = []\n",
    "\n",
    "for places in places_set:\n",
    "    coordinates = []\n",
    "    for place in places:\n",
    "        ll = geonames_query(place)\n",
    "        if ll != ('',''):\n",
    "            places_list.append(place)\n",
    "            coordinates.append(ll)\n",
    "    coordinates_list.append(coordinates)    \n",
    "\n",
    "print(coordinates_list[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert coordinates to float\n",
    "\n",
    "coordinates_list = [[(float(lat), float(long)) for lat, long in item] for item in coordinates_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Athens', 'Corinth', 'Greece']\n",
      "[(37.97945, 23.71622), (33.15401, -97.06473), (43.20978, -77.69306)]\n"
     ]
    }
   ],
   "source": [
    "# Get sample\n",
    "\n",
    "print(places_list)\n",
    "print(coordinates_list[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><iframe src=\"data:text/html;charset=utf-8;base64,PCFET0NUWVBFIGh0bWw+CjxoZWFkPiAgICAKICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgogICAgPHNjcmlwdD5MX1BSRUZFUl9DQU5WQVMgPSBmYWxzZTsgTF9OT19UT1VDSCA9IGZhbHNlOyBMX0RJU0FCTEVfM0QgPSBmYWxzZTs8L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2xlYWZsZXRAMS4yLjAvZGlzdC9sZWFmbGV0LmpzIj48L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2FqYXguZ29vZ2xlYXBpcy5jb20vYWpheC9saWJzL2pxdWVyeS8xLjExLjEvanF1ZXJ5Lm1pbi5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvanMvYm9vdHN0cmFwLm1pbi5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuanMiPjwvc2NyaXB0PgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2xlYWZsZXRAMS4yLjAvZGlzdC9sZWFmbGV0LmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvY3NzL2Jvb3RzdHJhcC5taW4uY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL21heGNkbi5ib290c3RyYXBjZG4uY29tL2Jvb3RzdHJhcC8zLjIuMC9jc3MvYm9vdHN0cmFwLXRoZW1lLm1pbi5jc3MiIC8+CiAgICA8bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9Imh0dHBzOi8vbWF4Y2RuLmJvb3RzdHJhcGNkbi5jb20vZm9udC1hd2Vzb21lLzQuNi4zL2Nzcy9mb250LWF3ZXNvbWUubWluLmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL3Jhd2dpdC5jb20vcHl0aG9uLXZpc3VhbGl6YXRpb24vZm9saXVtL21hc3Rlci9mb2xpdW0vdGVtcGxhdGVzL2xlYWZsZXQuYXdlc29tZS5yb3RhdGUuY3NzIiAvPgogICAgPHN0eWxlPmh0bWwsIGJvZHkge3dpZHRoOiAxMDAlO2hlaWdodDogMTAwJTttYXJnaW46IDA7cGFkZGluZzogMDt9PC9zdHlsZT4KICAgIDxzdHlsZT4jbWFwIHtwb3NpdGlvbjphYnNvbHV0ZTt0b3A6MDtib3R0b206MDtyaWdodDowO2xlZnQ6MDt9PC9zdHlsZT4KICAgIAogICAgICAgICAgICA8c3R5bGU+ICNtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEgewogICAgICAgICAgICAgICAgcG9zaXRpb24gOiByZWxhdGl2ZTsKICAgICAgICAgICAgICAgIHdpZHRoIDogOTYwLjBweDsKICAgICAgICAgICAgICAgIGhlaWdodDogNTEyLjBweDsKICAgICAgICAgICAgICAgIGxlZnQ6IDAuMCU7CiAgICAgICAgICAgICAgICB0b3A6IDAuMCU7CiAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgIDwvc3R5bGU+CiAgICAgICAgCjwvaGVhZD4KPGJvZHk+ICAgIAogICAgCiAgICAgICAgICAgIDxkaXYgY2xhc3M9ImZvbGl1bS1tYXAiIGlkPSJtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEiID48L2Rpdj4KICAgICAgICAKPC9ib2R5Pgo8c2NyaXB0PiAgICAKICAgIAoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgYm91bmRzID0gbnVsbDsKICAgICAgICAgICAgCgogICAgICAgICAgICB2YXIgbWFwX2VhYjdlOTQ4ZWQwYjRlNWZiMjVlMjE4NDQ5NTI0NTRhID0gTC5tYXAoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbWFwX2VhYjdlOTQ4ZWQwYjRlNWZiMjVlMjE4NDQ5NTI0NTRhJywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHtjZW50ZXI6IFszNy45Nzk0NSwyMy43MTYyMl0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB6b29tOiA0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4Qm91bmRzOiBib3VuZHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsYXllcnM6IFtdLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd29ybGRDb3B5SnVtcDogZmFsc2UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjcnM6IEwuQ1JTLkVQU0czODU3CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH0pOwogICAgICAgICAgICAKICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgdGlsZV9sYXllcl83NmIwMGUwOWM2Njc0MDgyYWI3YTQzYjhhNmNlY2FkZiA9IEwudGlsZUxheWVyKAogICAgICAgICAgICAgICAgJ2h0dHBzOi8vY2FydG9kYi1iYXNlbWFwcy17c30uZ2xvYmFsLnNzbC5mYXN0bHkubmV0L2xpZ2h0X2FsbC97en0ve3h9L3t5fS5wbmcnLAogICAgICAgICAgICAgICAgewogICJhdHRyaWJ1dGlvbiI6IG51bGwsCiAgImRldGVjdFJldGluYSI6IGZhbHNlLAogICJtYXhab29tIjogMTgsCiAgIm1pblpvb20iOiAxLAogICJub1dyYXAiOiBmYWxzZSwKICAic3ViZG9tYWlucyI6ICJhYmMiCn0KICAgICAgICAgICAgICAgICkuYWRkVG8obWFwX2VhYjdlOTQ4ZWQwYjRlNWZiMjVlMjE4NDQ5NTI0NTRhKTsKICAgICAgICAKICAgIAoKICAgICAgICAgICAgdmFyIG1hcmtlcl9jOWYzZGQ3YTM0Zjc0ZWFhYjk5OTM2NzFlYTYwOGY4NSA9IEwubWFya2VyKAogICAgICAgICAgICAgICAgWzM3Ljk3OTQ1LDIzLjcxNjIyXSwKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBpY29uOiBuZXcgTC5JY29uLkRlZmF1bHQoKQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIC5hZGRUbyhtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEpOwogICAgICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgcG9wdXBfNjkwMGM4NGUzNGFjNGI1ZDgyZDUxYjk2YjI1ZjRiMWMgPSBMLnBvcHVwKHttYXhXaWR0aDogJzMwMCd9KTsKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIGh0bWxfYWU2ZTgyYzZjNGY4NDBhOTk5NjI3MDllNDEwZGQzZTYgPSAkKCc8ZGl2IGlkPSJodG1sX2FlNmU4MmM2YzRmODQwYTk5OTYyNzA5ZTQxMGRkM2U2IiBzdHlsZT0id2lkdGg6IDEwMC4wJTsgaGVpZ2h0OiAxMDAuMCU7Ij5BdGhlbnMgMzcuOTc5NDUyMy43MTYyMjwvZGl2PicpWzBdOwogICAgICAgICAgICAgICAgcG9wdXBfNjkwMGM4NGUzNGFjNGI1ZDgyZDUxYjk2YjI1ZjRiMWMuc2V0Q29udGVudChodG1sX2FlNmU4MmM2YzRmODQwYTk5OTYyNzA5ZTQxMGRkM2U2KTsKICAgICAgICAgICAgCgogICAgICAgICAgICBtYXJrZXJfYzlmM2RkN2EzNGY3NGVhYWI5OTkzNjcxZWE2MDhmODUuYmluZFBvcHVwKHBvcHVwXzY5MDBjODRlMzRhYzRiNWQ4MmQ1MWI5NmIyNWY0YjFjKTsKCiAgICAgICAgICAgIAogICAgICAgIAogICAgCgogICAgICAgICAgICB2YXIgbWFya2VyXzY4NTU3YTgyNGExOTRhMTA5YmVhZDUwNzNhYjIzNzdlID0gTC5tYXJrZXIoCiAgICAgICAgICAgICAgICBbMzMuMTU0MDEsLTk3LjA2NDczXSwKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBpY29uOiBuZXcgTC5JY29uLkRlZmF1bHQoKQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIC5hZGRUbyhtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEpOwogICAgICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgcG9wdXBfYjUyYmFkNjk3NTIxNDc4M2FkYmI5OWE0NzNjOWEzYzAgPSBMLnBvcHVwKHttYXhXaWR0aDogJzMwMCd9KTsKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIGh0bWxfMGQ5ZWI2MmExMWY5NDIzMWIxZTNlODM0MGY0ZmFhNjAgPSAkKCc8ZGl2IGlkPSJodG1sXzBkOWViNjJhMTFmOTQyMzFiMWUzZTgzNDBmNGZhYTYwIiBzdHlsZT0id2lkdGg6IDEwMC4wJTsgaGVpZ2h0OiAxMDAuMCU7Ij5Db3JpbnRoIDMzLjE1NDAxLTk3LjA2NDczPC9kaXY+JylbMF07CiAgICAgICAgICAgICAgICBwb3B1cF9iNTJiYWQ2OTc1MjE0NzgzYWRiYjk5YTQ3M2M5YTNjMC5zZXRDb250ZW50KGh0bWxfMGQ5ZWI2MmExMWY5NDIzMWIxZTNlODM0MGY0ZmFhNjApOwogICAgICAgICAgICAKCiAgICAgICAgICAgIG1hcmtlcl82ODU1N2E4MjRhMTk0YTEwOWJlYWQ1MDczYWIyMzc3ZS5iaW5kUG9wdXAocG9wdXBfYjUyYmFkNjk3NTIxNDc4M2FkYmI5OWE0NzNjOWEzYzApOwoKICAgICAgICAgICAgCiAgICAgICAgCiAgICAKCiAgICAgICAgICAgIHZhciBtYXJrZXJfYTc3MTM1MDIwODJjNDlmYjk4ZTdkYzRjYjczZWE3OWQgPSBMLm1hcmtlcigKICAgICAgICAgICAgICAgIFs0My4yMDk3OCwtNzcuNjkzMDZdLAogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgIGljb246IG5ldyBMLkljb24uRGVmYXVsdCgpCiAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgLmFkZFRvKG1hcF9lYWI3ZTk0OGVkMGI0ZTVmYjI1ZTIxODQ0OTUyNDU0YSk7CiAgICAgICAgICAgIAogICAgCiAgICAgICAgICAgIHZhciBwb3B1cF81NzhjYjcxMzBmMTE0YTIyODljZDM5ZjM2ZmY1NTU1MyA9IEwucG9wdXAoe21heFdpZHRoOiAnMzAwJ30pOwoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgaHRtbF9kZDEyNGQ4YjRjMDU0NjM3YmZmZThiNTNjM2VmY2Y4MyA9ICQoJzxkaXYgaWQ9Imh0bWxfZGQxMjRkOGI0YzA1NDYzN2JmZmU4YjUzYzNlZmNmODMiIHN0eWxlPSJ3aWR0aDogMTAwLjAlOyBoZWlnaHQ6IDEwMC4wJTsiPkdyZWVjZSA0My4yMDk3OC03Ny42OTMwNjwvZGl2PicpWzBdOwogICAgICAgICAgICAgICAgcG9wdXBfNTc4Y2I3MTMwZjExNGEyMjg5Y2QzOWYzNmZmNTU1NTMuc2V0Q29udGVudChodG1sX2RkMTI0ZDhiNGMwNTQ2MzdiZmZlOGI1M2MzZWZjZjgzKTsKICAgICAgICAgICAgCgogICAgICAgICAgICBtYXJrZXJfYTc3MTM1MDIwODJjNDlmYjk4ZTdkYzRjYjczZWE3OWQuYmluZFBvcHVwKHBvcHVwXzU3OGNiNzEzMGYxMTRhMjI4OWNkMzlmMzZmZjU1NTUzKTsKCiAgICAgICAgICAgIAogICAgICAgIAo8L3NjcmlwdD4=\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
      ],
      "text/plain": [
       "<folium.folium.Map at 0x10aabf4e0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set up Folium and populate with coordinates\n",
    "\n",
    "basemap = folium.Map(location=[37.97945, 23.71622], zoom_start=4, tiles='cartodbpositron', width=960, height=512)\n",
    "\n",
    "for i, c in enumerate(coordinates_list[0]):\n",
    "    folium.Marker([c[0], c[1]], popup='{} {}{}'.format(places_list[i], c[0], c[1])).add_to(basemap)\n",
    "    \n",
    "basemap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><iframe src=\"data:text/html;charset=utf-8;base64,PCFET0NUWVBFIGh0bWw+CjxoZWFkPiAgICAKICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgogICAgPHNjcmlwdD5MX1BSRUZFUl9DQU5WQVMgPSBmYWxzZTsgTF9OT19UT1VDSCA9IGZhbHNlOyBMX0RJU0FCTEVfM0QgPSBmYWxzZTs8L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2xlYWZsZXRAMS4yLjAvZGlzdC9sZWFmbGV0LmpzIj48L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2FqYXguZ29vZ2xlYXBpcy5jb20vYWpheC9saWJzL2pxdWVyeS8xLjExLjEvanF1ZXJ5Lm1pbi5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvanMvYm9vdHN0cmFwLm1pbi5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuanMiPjwvc2NyaXB0PgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2xlYWZsZXRAMS4yLjAvZGlzdC9sZWFmbGV0LmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvY3NzL2Jvb3RzdHJhcC5taW4uY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL21heGNkbi5ib290c3RyYXBjZG4uY29tL2Jvb3RzdHJhcC8zLjIuMC9jc3MvYm9vdHN0cmFwLXRoZW1lLm1pbi5jc3MiIC8+CiAgICA8bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9Imh0dHBzOi8vbWF4Y2RuLmJvb3RzdHJhcGNkbi5jb20vZm9udC1hd2Vzb21lLzQuNi4zL2Nzcy9mb250LWF3ZXNvbWUubWluLmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL3Jhd2dpdC5jb20vcHl0aG9uLXZpc3VhbGl6YXRpb24vZm9saXVtL21hc3Rlci9mb2xpdW0vdGVtcGxhdGVzL2xlYWZsZXQuYXdlc29tZS5yb3RhdGUuY3NzIiAvPgogICAgPHN0eWxlPmh0bWwsIGJvZHkge3dpZHRoOiAxMDAlO2hlaWdodDogMTAwJTttYXJnaW46IDA7cGFkZGluZzogMDt9PC9zdHlsZT4KICAgIDxzdHlsZT4jbWFwIHtwb3NpdGlvbjphYnNvbHV0ZTt0b3A6MDtib3R0b206MDtyaWdodDowO2xlZnQ6MDt9PC9zdHlsZT4KICAgIAogICAgICAgICAgICA8c3R5bGU+ICNtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEgewogICAgICAgICAgICAgICAgcG9zaXRpb24gOiByZWxhdGl2ZTsKICAgICAgICAgICAgICAgIHdpZHRoIDogOTYwLjBweDsKICAgICAgICAgICAgICAgIGhlaWdodDogNTEyLjBweDsKICAgICAgICAgICAgICAgIGxlZnQ6IDAuMCU7CiAgICAgICAgICAgICAgICB0b3A6IDAuMCU7CiAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgIDwvc3R5bGU+CiAgICAgICAgCjwvaGVhZD4KPGJvZHk+ICAgIAogICAgCiAgICAgICAgICAgIDxkaXYgY2xhc3M9ImZvbGl1bS1tYXAiIGlkPSJtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEiID48L2Rpdj4KICAgICAgICAKPC9ib2R5Pgo8c2NyaXB0PiAgICAKICAgIAoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgYm91bmRzID0gbnVsbDsKICAgICAgICAgICAgCgogICAgICAgICAgICB2YXIgbWFwX2VhYjdlOTQ4ZWQwYjRlNWZiMjVlMjE4NDQ5NTI0NTRhID0gTC5tYXAoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbWFwX2VhYjdlOTQ4ZWQwYjRlNWZiMjVlMjE4NDQ5NTI0NTRhJywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHtjZW50ZXI6IFszNy45Nzk0NSwyMy43MTYyMl0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB6b29tOiAyLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4Qm91bmRzOiBib3VuZHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsYXllcnM6IFtdLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd29ybGRDb3B5SnVtcDogZmFsc2UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjcnM6IEwuQ1JTLkVQU0czODU3CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH0pOwogICAgICAgICAgICAKICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgdGlsZV9sYXllcl83NmIwMGUwOWM2Njc0MDgyYWI3YTQzYjhhNmNlY2FkZiA9IEwudGlsZUxheWVyKAogICAgICAgICAgICAgICAgJ2h0dHBzOi8vY2FydG9kYi1iYXNlbWFwcy17c30uZ2xvYmFsLnNzbC5mYXN0bHkubmV0L2xpZ2h0X2FsbC97en0ve3h9L3t5fS5wbmcnLAogICAgICAgICAgICAgICAgewogICJhdHRyaWJ1dGlvbiI6IG51bGwsCiAgImRldGVjdFJldGluYSI6IGZhbHNlLAogICJtYXhab29tIjogMTgsCiAgIm1pblpvb20iOiAxLAogICJub1dyYXAiOiBmYWxzZSwKICAic3ViZG9tYWlucyI6ICJhYmMiCn0KICAgICAgICAgICAgICAgICkuYWRkVG8obWFwX2VhYjdlOTQ4ZWQwYjRlNWZiMjVlMjE4NDQ5NTI0NTRhKTsKICAgICAgICAKICAgIAoKICAgICAgICAgICAgdmFyIG1hcmtlcl9jOWYzZGQ3YTM0Zjc0ZWFhYjk5OTM2NzFlYTYwOGY4NSA9IEwubWFya2VyKAogICAgICAgICAgICAgICAgWzM3Ljk3OTQ1LDIzLjcxNjIyXSwKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBpY29uOiBuZXcgTC5JY29uLkRlZmF1bHQoKQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIC5hZGRUbyhtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEpOwogICAgICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgcG9wdXBfNjkwMGM4NGUzNGFjNGI1ZDgyZDUxYjk2YjI1ZjRiMWMgPSBMLnBvcHVwKHttYXhXaWR0aDogJzMwMCd9KTsKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIGh0bWxfYWU2ZTgyYzZjNGY4NDBhOTk5NjI3MDllNDEwZGQzZTYgPSAkKCc8ZGl2IGlkPSJodG1sX2FlNmU4MmM2YzRmODQwYTk5OTYyNzA5ZTQxMGRkM2U2IiBzdHlsZT0id2lkdGg6IDEwMC4wJTsgaGVpZ2h0OiAxMDAuMCU7Ij5BdGhlbnMgMzcuOTc5NDUyMy43MTYyMjwvZGl2PicpWzBdOwogICAgICAgICAgICAgICAgcG9wdXBfNjkwMGM4NGUzNGFjNGI1ZDgyZDUxYjk2YjI1ZjRiMWMuc2V0Q29udGVudChodG1sX2FlNmU4MmM2YzRmODQwYTk5OTYyNzA5ZTQxMGRkM2U2KTsKICAgICAgICAgICAgCgogICAgICAgICAgICBtYXJrZXJfYzlmM2RkN2EzNGY3NGVhYWI5OTkzNjcxZWE2MDhmODUuYmluZFBvcHVwKHBvcHVwXzY5MDBjODRlMzRhYzRiNWQ4MmQ1MWI5NmIyNWY0YjFjKTsKCiAgICAgICAgICAgIAogICAgICAgIAogICAgCgogICAgICAgICAgICB2YXIgbWFya2VyXzY4NTU3YTgyNGExOTRhMTA5YmVhZDUwNzNhYjIzNzdlID0gTC5tYXJrZXIoCiAgICAgICAgICAgICAgICBbMzMuMTU0MDEsLTk3LjA2NDczXSwKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBpY29uOiBuZXcgTC5JY29uLkRlZmF1bHQoKQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIC5hZGRUbyhtYXBfZWFiN2U5NDhlZDBiNGU1ZmIyNWUyMTg0NDk1MjQ1NGEpOwogICAgICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgcG9wdXBfYjUyYmFkNjk3NTIxNDc4M2FkYmI5OWE0NzNjOWEzYzAgPSBMLnBvcHVwKHttYXhXaWR0aDogJzMwMCd9KTsKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIGh0bWxfMGQ5ZWI2MmExMWY5NDIzMWIxZTNlODM0MGY0ZmFhNjAgPSAkKCc8ZGl2IGlkPSJodG1sXzBkOWViNjJhMTFmOTQyMzFiMWUzZTgzNDBmNGZhYTYwIiBzdHlsZT0id2lkdGg6IDEwMC4wJTsgaGVpZ2h0OiAxMDAuMCU7Ij5Db3JpbnRoIDMzLjE1NDAxLTk3LjA2NDczPC9kaXY+JylbMF07CiAgICAgICAgICAgICAgICBwb3B1cF9iNTJiYWQ2OTc1MjE0NzgzYWRiYjk5YTQ3M2M5YTNjMC5zZXRDb250ZW50KGh0bWxfMGQ5ZWI2MmExMWY5NDIzMWIxZTNlODM0MGY0ZmFhNjApOwogICAgICAgICAgICAKCiAgICAgICAgICAgIG1hcmtlcl82ODU1N2E4MjRhMTk0YTEwOWJlYWQ1MDczYWIyMzc3ZS5iaW5kUG9wdXAocG9wdXBfYjUyYmFkNjk3NTIxNDc4M2FkYmI5OWE0NzNjOWEzYzApOwoKICAgICAgICAgICAgCiAgICAgICAgCiAgICAKCiAgICAgICAgICAgIHZhciBtYXJrZXJfYTc3MTM1MDIwODJjNDlmYjk4ZTdkYzRjYjczZWE3OWQgPSBMLm1hcmtlcigKICAgICAgICAgICAgICAgIFs0My4yMDk3OCwtNzcuNjkzMDZdLAogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgIGljb246IG5ldyBMLkljb24uRGVmYXVsdCgpCiAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgLmFkZFRvKG1hcF9lYWI3ZTk0OGVkMGI0ZTVmYjI1ZTIxODQ0OTUyNDU0YSk7CiAgICAgICAgICAgIAogICAgCiAgICAgICAgICAgIHZhciBwb3B1cF81NzhjYjcxMzBmMTE0YTIyODljZDM5ZjM2ZmY1NTU1MyA9IEwucG9wdXAoe21heFdpZHRoOiAnMzAwJ30pOwoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgaHRtbF9kZDEyNGQ4YjRjMDU0NjM3YmZmZThiNTNjM2VmY2Y4MyA9ICQoJzxkaXYgaWQ9Imh0bWxfZGQxMjRkOGI0YzA1NDYzN2JmZmU4YjUzYzNlZmNmODMiIHN0eWxlPSJ3aWR0aDogMTAwLjAlOyBoZWlnaHQ6IDEwMC4wJTsiPkdyZWVjZSA0My4yMDk3OC03Ny42OTMwNjwvZGl2PicpWzBdOwogICAgICAgICAgICAgICAgcG9wdXBfNTc4Y2I3MTMwZjExNGEyMjg5Y2QzOWYzNmZmNTU1NTMuc2V0Q29udGVudChodG1sX2RkMTI0ZDhiNGMwNTQ2MzdiZmZlOGI1M2MzZWZjZjgzKTsKICAgICAgICAgICAgCgogICAgICAgICAgICBtYXJrZXJfYTc3MTM1MDIwODJjNDlmYjk4ZTdkYzRjYjczZWE3OWQuYmluZFBvcHVwKHBvcHVwXzU3OGNiNzEzMGYxMTRhMjI4OWNkMzlmMzZmZjU1NTUzKTsKCiAgICAgICAgICAgIAogICAgICAgIAo8L3NjcmlwdD4=\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
      ],
      "text/plain": [
       "<folium.folium.Map at 0x10aabf4e0>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Why does only Athens return a result on the map? Check zoom...\n",
    "\n",
    "basemap.zoom_start = 2\n",
    "basemap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Disambiguate placenames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# So we have learned that we can't use the top 'relevance' score from the Geoname API, or we\n",
    "# get results like 'Corinth, Texas' and 'Greece, NY'. We need some plan for disambiguating the\n",
    "# Geonames results..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import geocoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'address': 'Corinth',\n",
      " 'class_description': 'country, state, region,...',\n",
      " 'code': 'ADM2',\n",
      " 'country': 'Saint Lucia',\n",
      " 'country_code': 'LC',\n",
      " 'description': 'second-order administrative division',\n",
      " 'feature_class': 'A',\n",
      " 'geonames_id': 11351423,\n",
      " 'lat': '14.0471',\n",
      " 'lng': '-60.96046',\n",
      " 'ok': True,\n",
      " 'population': 1382,\n",
      " 'raw': {'adminCode1': '06',\n",
      "         'adminCodes1': {'ISO3166_2': '06'},\n",
      "         'adminName1': 'Gros-Islet',\n",
      "         'countryCode': 'LC',\n",
      "         'countryId': '3576468',\n",
      "         'countryName': 'Saint Lucia',\n",
      "         'fcl': 'A',\n",
      "         'fclName': 'country, state, region,...',\n",
      "         'fcode': 'ADM2',\n",
      "         'fcodeName': 'second-order administrative division',\n",
      "         'geonameId': 11351423,\n",
      "         'lat': '14.0471',\n",
      "         'lng': '-60.96046',\n",
      "         'name': 'Corinth',\n",
      "         'population': 1382,\n",
      "         'toponymName': 'Corinth'},\n",
      " 'state': 'Gros-Islet',\n",
      " 'state_code': '06',\n",
      " 'status': 'OK'}\n"
     ]
    }
   ],
   "source": [
    "# Retrieve json from geonames API (for fun this time using geocoder)\n",
    "\n",
    "geocoder_results = []\n",
    "\n",
    "for place in places_list:\n",
    "    results = geocoder.geonames(place, maxRows=5, key=USERNAME)\n",
    "    jsons = []\n",
    "    for result in results:\n",
    "        jsons.append(result.json)\n",
    "    geocoder_results.append(jsons)\n",
    "    \n",
    "pprint(geocoder_results[1][0]) # Corinth, but a wrong Corinth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list of 'country' from the geonames json results\n",
    "\n",
    "countries = []\n",
    "\n",
    "for results in geocoder_results:\n",
    "    for item in results:\n",
    "        countries.append(item['country'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Greece\n"
     ]
    }
   ],
   "source": [
    "# Determine which country appears most often\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "top_country = sorted(Counter(countries))[0]\n",
    "print(top_country)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Athens', 'Corinth', 'Greece']\n",
      "[(37.97945, 23.71622), (37.94007, 22.9513), (39.0, 22.0)]\n"
     ]
    }
   ],
   "source": [
    "# Iterate over geocoder_results and keep the first lat/long that matches the top country\n",
    "\n",
    "coordinates = []\n",
    "\n",
    "for i, results in enumerate(geocoder_results):\n",
    "    for item in results:\n",
    "        if item['country'] == top_country:\n",
    "            coordinates.append((float(item['lat']), float(item['lng'])))\n",
    "            break # Only get the first item for now\n",
    "\n",
    "print(places_list)            \n",
    "print(coordinates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create revised map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Map of relevant locations in Broneer et al.'s \"Ancient Corinth: A guide to the excavations\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"width:100%;\"><div style=\"position:relative;width:100%;height:0;padding-bottom:60%;\"><iframe src=\"data:text/html;charset=utf-8;base64,PCFET0NUWVBFIGh0bWw+CjxoZWFkPiAgICAKICAgIDxtZXRhIGh0dHAtZXF1aXY9ImNvbnRlbnQtdHlwZSIgY29udGVudD0idGV4dC9odG1sOyBjaGFyc2V0PVVURi04IiAvPgogICAgPHNjcmlwdD5MX1BSRUZFUl9DQU5WQVMgPSBmYWxzZTsgTF9OT19UT1VDSCA9IGZhbHNlOyBMX0RJU0FCTEVfM0QgPSBmYWxzZTs8L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2xlYWZsZXRAMS4yLjAvZGlzdC9sZWFmbGV0LmpzIj48L3NjcmlwdD4KICAgIDxzY3JpcHQgc3JjPSJodHRwczovL2FqYXguZ29vZ2xlYXBpcy5jb20vYWpheC9saWJzL2pxdWVyeS8xLjExLjEvanF1ZXJ5Lm1pbi5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvanMvYm9vdHN0cmFwLm1pbi5qcyI+PC9zY3JpcHQ+CiAgICA8c2NyaXB0IHNyYz0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuanMiPjwvc2NyaXB0PgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL2Nkbi5qc2RlbGl2ci5uZXQvbnBtL2xlYWZsZXRAMS4yLjAvZGlzdC9sZWFmbGV0LmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9tYXhjZG4uYm9vdHN0cmFwY2RuLmNvbS9ib290c3RyYXAvMy4yLjAvY3NzL2Jvb3RzdHJhcC5taW4uY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL21heGNkbi5ib290c3RyYXBjZG4uY29tL2Jvb3RzdHJhcC8zLjIuMC9jc3MvYm9vdHN0cmFwLXRoZW1lLm1pbi5jc3MiIC8+CiAgICA8bGluayByZWw9InN0eWxlc2hlZXQiIGhyZWY9Imh0dHBzOi8vbWF4Y2RuLmJvb3RzdHJhcGNkbi5jb20vZm9udC1hd2Vzb21lLzQuNi4zL2Nzcy9mb250LWF3ZXNvbWUubWluLmNzcyIgLz4KICAgIDxsaW5rIHJlbD0ic3R5bGVzaGVldCIgaHJlZj0iaHR0cHM6Ly9jZG5qcy5jbG91ZGZsYXJlLmNvbS9hamF4L2xpYnMvTGVhZmxldC5hd2Vzb21lLW1hcmtlcnMvMi4wLjIvbGVhZmxldC5hd2Vzb21lLW1hcmtlcnMuY3NzIiAvPgogICAgPGxpbmsgcmVsPSJzdHlsZXNoZWV0IiBocmVmPSJodHRwczovL3Jhd2dpdC5jb20vcHl0aG9uLXZpc3VhbGl6YXRpb24vZm9saXVtL21hc3Rlci9mb2xpdW0vdGVtcGxhdGVzL2xlYWZsZXQuYXdlc29tZS5yb3RhdGUuY3NzIiAvPgogICAgPHN0eWxlPmh0bWwsIGJvZHkge3dpZHRoOiAxMDAlO2hlaWdodDogMTAwJTttYXJnaW46IDA7cGFkZGluZzogMDt9PC9zdHlsZT4KICAgIDxzdHlsZT4jbWFwIHtwb3NpdGlvbjphYnNvbHV0ZTt0b3A6MDtib3R0b206MDtyaWdodDowO2xlZnQ6MDt9PC9zdHlsZT4KICAgIAogICAgICAgICAgICA8c3R5bGU+ICNtYXBfYTkyYzg1ZDNkOTJiNGE3YWJiZDAzZTFjMjRhNjM5MGQgewogICAgICAgICAgICAgICAgcG9zaXRpb24gOiByZWxhdGl2ZTsKICAgICAgICAgICAgICAgIHdpZHRoIDogOTYwLjBweDsKICAgICAgICAgICAgICAgIGhlaWdodDogNTEyLjBweDsKICAgICAgICAgICAgICAgIGxlZnQ6IDAuMCU7CiAgICAgICAgICAgICAgICB0b3A6IDAuMCU7CiAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgIDwvc3R5bGU+CiAgICAgICAgCjwvaGVhZD4KPGJvZHk+ICAgIAogICAgCiAgICAgICAgICAgIDxkaXYgY2xhc3M9ImZvbGl1bS1tYXAiIGlkPSJtYXBfYTkyYzg1ZDNkOTJiNGE3YWJiZDAzZTFjMjRhNjM5MGQiID48L2Rpdj4KICAgICAgICAKPC9ib2R5Pgo8c2NyaXB0PiAgICAKICAgIAoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgYm91bmRzID0gbnVsbDsKICAgICAgICAgICAgCgogICAgICAgICAgICB2YXIgbWFwX2E5MmM4NWQzZDkyYjRhN2FiYmQwM2UxYzI0YTYzOTBkID0gTC5tYXAoCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAnbWFwX2E5MmM4NWQzZDkyYjRhN2FiYmQwM2UxYzI0YTYzOTBkJywKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHtjZW50ZXI6IFszNy45Nzk0NSwyMy43MTYyMl0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB6b29tOiA2LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4Qm91bmRzOiBib3VuZHMsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBsYXllcnM6IFtdLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgd29ybGRDb3B5SnVtcDogZmFsc2UsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjcnM6IEwuQ1JTLkVQU0czODU3CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH0pOwogICAgICAgICAgICAKICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgdGlsZV9sYXllcl9jMmVhMGVhNWMyYmM0ZWRhOWM3M2VmOWYwMzU3YmU4OCA9IEwudGlsZUxheWVyKAogICAgICAgICAgICAgICAgJ2h0dHBzOi8vY2FydG9kYi1iYXNlbWFwcy17c30uZ2xvYmFsLnNzbC5mYXN0bHkubmV0L2xpZ2h0X2FsbC97en0ve3h9L3t5fS5wbmcnLAogICAgICAgICAgICAgICAgewogICJhdHRyaWJ1dGlvbiI6IG51bGwsCiAgImRldGVjdFJldGluYSI6IGZhbHNlLAogICJtYXhab29tIjogMTgsCiAgIm1pblpvb20iOiAxLAogICJub1dyYXAiOiBmYWxzZSwKICAic3ViZG9tYWlucyI6ICJhYmMiCn0KICAgICAgICAgICAgICAgICkuYWRkVG8obWFwX2E5MmM4NWQzZDkyYjRhN2FiYmQwM2UxYzI0YTYzOTBkKTsKICAgICAgICAKICAgIAoKICAgICAgICAgICAgdmFyIG1hcmtlcl9iZmQwMTc3Y2QxZjE0MjczOTllNDdlMTBjMjc0YTZiOCA9IEwubWFya2VyKAogICAgICAgICAgICAgICAgWzM3Ljk3OTQ1LDIzLjcxNjIyXSwKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBpY29uOiBuZXcgTC5JY29uLkRlZmF1bHQoKQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIC5hZGRUbyhtYXBfYTkyYzg1ZDNkOTJiNGE3YWJiZDAzZTFjMjRhNjM5MGQpOwogICAgICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgcG9wdXBfMzhiMzY0MzQyMzFhNDc2NTgzODljNjFmNTI2ODU2YTAgPSBMLnBvcHVwKHttYXhXaWR0aDogJzMwMCd9KTsKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIGh0bWxfOGM1ZDI3MTQ0N2U5NDdkZDlhYzU1NDg0ZjVhNjAwNWUgPSAkKCc8ZGl2IGlkPSJodG1sXzhjNWQyNzE0NDdlOTQ3ZGQ5YWM1NTQ4NGY1YTYwMDVlIiBzdHlsZT0id2lkdGg6IDEwMC4wJTsgaGVpZ2h0OiAxMDAuMCU7Ij5BdGhlbnMgKDM3Ljk3OTQ1LCAyMy43MTYyMik8L2Rpdj4nKVswXTsKICAgICAgICAgICAgICAgIHBvcHVwXzM4YjM2NDM0MjMxYTQ3NjU4Mzg5YzYxZjUyNjg1NmEwLnNldENvbnRlbnQoaHRtbF84YzVkMjcxNDQ3ZTk0N2RkOWFjNTU0ODRmNWE2MDA1ZSk7CiAgICAgICAgICAgIAoKICAgICAgICAgICAgbWFya2VyX2JmZDAxNzdjZDFmMTQyNzM5OWU0N2UxMGMyNzRhNmI4LmJpbmRQb3B1cChwb3B1cF8zOGIzNjQzNDIzMWE0NzY1ODM4OWM2MWY1MjY4NTZhMCk7CgogICAgICAgICAgICAKICAgICAgICAKICAgIAoKICAgICAgICAgICAgdmFyIG1hcmtlcl8zOTEzOWFiYTEwZDU0ZWMzYjQwNzU0MWJkZjgxMjUzYiA9IEwubWFya2VyKAogICAgICAgICAgICAgICAgWzM3Ljk0MDA3LDIyLjk1MTNdLAogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgIGljb246IG5ldyBMLkljb24uRGVmYXVsdCgpCiAgICAgICAgICAgICAgICAgICAgfQogICAgICAgICAgICAgICAgKQogICAgICAgICAgICAgICAgLmFkZFRvKG1hcF9hOTJjODVkM2Q5MmI0YTdhYmJkMDNlMWMyNGE2MzkwZCk7CiAgICAgICAgICAgIAogICAgCiAgICAgICAgICAgIHZhciBwb3B1cF8xYWI2MTg1MzI0YzI0Y2VmYTE2NWM5OTNkN2JlZjIwZiA9IEwucG9wdXAoe21heFdpZHRoOiAnMzAwJ30pOwoKICAgICAgICAgICAgCiAgICAgICAgICAgICAgICB2YXIgaHRtbF80ZjM2OWI3ZTUxN2Y0ZDU2YTZjZjcwNTg3NWFlZTY3OSA9ICQoJzxkaXYgaWQ9Imh0bWxfNGYzNjliN2U1MTdmNGQ1NmE2Y2Y3MDU4NzVhZWU2NzkiIHN0eWxlPSJ3aWR0aDogMTAwLjAlOyBoZWlnaHQ6IDEwMC4wJTsiPkNvcmludGggKDM3Ljk0MDA3LCAyMi45NTEzKTwvZGl2PicpWzBdOwogICAgICAgICAgICAgICAgcG9wdXBfMWFiNjE4NTMyNGMyNGNlZmExNjVjOTkzZDdiZWYyMGYuc2V0Q29udGVudChodG1sXzRmMzY5YjdlNTE3ZjRkNTZhNmNmNzA1ODc1YWVlNjc5KTsKICAgICAgICAgICAgCgogICAgICAgICAgICBtYXJrZXJfMzkxMzlhYmExMGQ1NGVjM2I0MDc1NDFiZGY4MTI1M2IuYmluZFBvcHVwKHBvcHVwXzFhYjYxODUzMjRjMjRjZWZhMTY1Yzk5M2Q3YmVmMjBmKTsKCiAgICAgICAgICAgIAogICAgICAgIAogICAgCgogICAgICAgICAgICB2YXIgbWFya2VyX2NmMWZlYWUyODBiMjQ1MDc5OWE5NzUwM2FkZTEzM2EwID0gTC5tYXJrZXIoCiAgICAgICAgICAgICAgICBbMzkuMCwyMi4wXSwKICAgICAgICAgICAgICAgIHsKICAgICAgICAgICAgICAgICAgICBpY29uOiBuZXcgTC5JY29uLkRlZmF1bHQoKQogICAgICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgICAgICkKICAgICAgICAgICAgICAgIC5hZGRUbyhtYXBfYTkyYzg1ZDNkOTJiNGE3YWJiZDAzZTFjMjRhNjM5MGQpOwogICAgICAgICAgICAKICAgIAogICAgICAgICAgICB2YXIgcG9wdXBfN2QyZDRlMTU5ZjhiNDEwMzg5ZTY3YzljOTAwYTZkZjcgPSBMLnBvcHVwKHttYXhXaWR0aDogJzMwMCd9KTsKCiAgICAgICAgICAgIAogICAgICAgICAgICAgICAgdmFyIGh0bWxfMmI0ZjQzNTYzNjZjNDAwNGIyNWQ3N2I3YjljZjA4MDUgPSAkKCc8ZGl2IGlkPSJodG1sXzJiNGY0MzU2MzY2YzQwMDRiMjVkNzdiN2I5Y2YwODA1IiBzdHlsZT0id2lkdGg6IDEwMC4wJTsgaGVpZ2h0OiAxMDAuMCU7Ij5HcmVlY2UgKDM5LjAsIDIyLjApPC9kaXY+JylbMF07CiAgICAgICAgICAgICAgICBwb3B1cF83ZDJkNGUxNTlmOGI0MTAzODllNjdjOWM5MDBhNmRmNy5zZXRDb250ZW50KGh0bWxfMmI0ZjQzNTYzNjZjNDAwNGIyNWQ3N2I3YjljZjA4MDUpOwogICAgICAgICAgICAKCiAgICAgICAgICAgIG1hcmtlcl9jZjFmZWFlMjgwYjI0NTA3OTlhOTc1MDNhZGUxMzNhMC5iaW5kUG9wdXAocG9wdXBfN2QyZDRlMTU5ZjhiNDEwMzg5ZTY3YzljOTAwYTZkZjcpOwoKICAgICAgICAgICAgCiAgICAgICAgCjwvc2NyaXB0Pg==\" style=\"position:absolute;width:100%;height:100%;left:0;top:0;border:none !important;\" allowfullscreen webkitallowfullscreen mozallowfullscreen></iframe></div></div>"
      ],
      "text/plain": [
       "<folium.folium.Map at 0x10dcb71d0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set up Folium and populate with 'disambiguated' coordinates\n",
    "\n",
    "basemap = folium.Map(location=[37.97945, 23.71622], zoom_start=6, tiles='cartodbpositron', width=960, height=512)\n",
    "\n",
    "for i, c in enumerate(coordinates):\n",
    "    folium.Marker([c[0], c[1]], popup='{} ({}, {})'.format(places_list[i], c[0], c[1])).add_to(basemap)\n",
    "\n",
    "print('Map of relevant locations in Broneer et al.\\'s \"Ancient Corinth: A guide to the excavations\"')\n",
    "basemap"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
